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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )□ Responsive to connmunication{s) filed on , 

2a)n This action is FINAL, 2b)l3 This action is non-final. 

3) n Since this application is in con(jition for allowance except for formal matters, prosecution as to the merits is 

closeci in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) 13 Claim(s) 1-12 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) 0 Claim(s) is/are allowed. 

6) 13 Claim(s) 1-5 and 7-12 is/are rejected. 

7) 13 Claim(s) 6 is/are objected to. 

8) 0 Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) 13 The specification is objected to by the Examiner. 

10) S The drawing(s) filed on 04 April 2001 is/are: a)n accepted or b)S objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the connection is required If the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) n The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) 0 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)n All b)n Some * 0)0 None of: 

1 .□ Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. . 

3. n Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
Drawings 

1 . The drawings are objected to as failing to comply with 37 CFR 1 .84(p)(4) 
because reference characters "100" and "599" have been used to designate two 
different items in Fig. 1 and Fig. 5, respectively. A proposed drawing correction or 
corrected drawings are required in reply to the Office action to avoid abandonment of 
the application. The objection to the drawings will not be held in abeyance. 

2. The drawings are objected to because blocks in figures should have descriptive 
as well as numeric labels. No descriptive text labels for the blocks are provided in Fig. 
1 . A proposed drawing correction or corrected drawings are required in reply to the 
Office action to avoid abandonment of the application. The objection to the drawings 
will not be held in abeyance. 

Specification 

3. The disclosure is objected to because of the following informalities: 

• P. 9, line 25 - P. 1 0, line 5: Lines 1 -5 of P. 1 0 do not describe how a volume 
(Pj) is assembled 

• Claim 1, line 6: "corresponding a volume" should have been "corresponding 
volume" 

• Claim 5, line 3: "are the marker" should have been "as the marker" 

• Claim 10, line 5: "less" appears to be extraneous 
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Appropriate correction is required. 

Claim Objections 

4. Claim 4 is objected to because it includes reference characters (102 and 210) 
which are not enclosed within parentheses. 

Reference characters corresponding to elements recited in the detailed 
description of the drawings and used in conjunction with the recitation of the same 
element or group of elements in the claims should be enclosed within parentheses so 
as to avoid confusion with other numbers or characters which may appear in the claims. 
See MPEP§ 608.01 (m). 

Appropriate correction is required. 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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6. Claims 1, 7, 12 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Wu (US 6,529.202), in view of Frisken et al. (US 6,603,484). 



Regarding claim 1 , Wu discloses the following: 

• assigning a feature vector to each pixel of the video 

[Col. 4, lines 1-3. Note that the color of the pixel is the feature vector] 

• identifying selected pixels of the video as marker pixels 
[Co. 3, lines 66-67] 

• assembling each marker pixel and pixels adjacent to the marker pixel into a 
corresponding a volume if the distance between the feature vector of the marker 
pixel and the feature vector of the adjacent pixels is less than a first 
predetermined threshold; 

[Col. 3, line 67 -Col. 4, line 6] 

• assigning a first score and descriptors to each volume 

[Fig. 2, numerals 203, 204; Fig. 3, numeral 303; Col. 4, lines 31-33; Col. 5, lines 
39-42. Note that the size of each region is the first score and the average color 
of the region is the descriptor] 

• sorting the volumes in a high-to-low order according to the first scores and 
processing the volumes in the high-to-low order 

[Fig. 3, numerals 303, 304; Col. 5, lines 39-42, 58-60] 

• comparing the descriptor of the volume to the descriptor of another volume to 
determine a second score; and 

[Fig. 2, numeral 205; Col. 4, lines 40-57. Note that D in the equation between 
lines 50 and 53 is the second score] 

• combining the volume with the adjacent volume if the second score passes a 
second threshold 

[Fig. 2, numeral 205; Col. 4, lines 54-57] 

• repeating the comparing and combining steps 
[Fig. 3, numerals 302-309] 

Wu fails to disclose the following, which Frisken et al. teaches: 

• to generate a video object in a multi-resolution video object tree 
[Fig. 14. Note that each node represents an object] 

• repeating until a single video representing the video remains 

[Fig. 12, numeral 1275, 1276, 1290. Col. 18, lines 14-17. Note that the root 
node represents the entire video (Fig. 3; Col. 9, lines 20-21)] 
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While Wu discloses comparing and combining regions (e.g., volumes) that are close in 
the color space [Col. 4, lines 41-44], Frisken et al. teaches that the comparison and 
combination be done on adjacent cells (e.g., volumes) [Col. 17, line 66 - Col. 18, line 2; 
note that by definition children cells are adjacent]. 

Therefore, it would have been obvious to one of ordinary skill at the time the invention 
was made to modify the invention of Wu by combining adjacent volumes to generate a 
multi-resolution object tree and terminating the process when only a single video 
representing the video remains as taught by Frisken et al. so that a spatially integral 
regions (e.g., volumes) can be constructed and an efficient data representation is 
produced to support detail-directed operations. 

7. Regarding claim 7, Wu further discloses 

• the feature vector is based on a color of the pixel 

[Col. 4, lines 1-3. Note that the color of the pixel is the feature vector] 

8. Claim 1 2 is similarly analyzed and rejected as per claim 1 . 

9. Claim 2 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wu (US 
6,529,202) and Frisken et al. (US 6,603,484) as applied to claims 1, 7, 12 further in view 
of Kimetal.(US 5,502,510). 

Regarding claim 2, the combined invention of Wu and Frisken et al. teaches everything 
except the following, which Kim et al. teach: 
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• each pixel has spatial (x,y) and tinne (t) coordinates to indicate a location of the 
pixel and the volumes in a spatial-temporal collocated overlapping scene of the 
video 

[Col. 3, lines 1-3] 

Therefore, it would have been obvious to one of ordinary skill at the time the invention 
was made to modify the combined invention of Wu and Frisken et al. by using (x, y, t) to 
designate the spatial-temporal coordinate of a pixel in a volume as taught by Kim (and 
actually well-known in the art) in order to be able to address the pixels that are to be 
operated upon. 



10. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wu (US 
6,529,202), Frisken et al, (US 6,603,484) and Kim et al. (US 5,502,510) as applied to 
claim 2, further in view of Kumar et al. (PUB No.: US 2001/0038718). 

Regarding claim 3, the combined invention of Wu and Frisken et al. and Kim teaches 
everything in its parent claim (claim 2), as well as that the video includes a plurality of 
frames. [Kim. Col. 3, lines 1-3. Note the temporal component implies a plurality of 
frames.] 

The combined invention of Wu and Frisken et al. and Kim fails to teach the following, 
which Kumar et al. teach: 
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• projecting a portion of each video object in a particular frame to intersect the 
projection of the video object in an adjacent frame to provide continuous 
silhouettes of the video object according to the time t coordinates 

[P. 3. Paragraph 0034, lines 8-10, 14-16] 

Therefore, it would have been obvious to one of ordinary skill at the time the invention 
was made to modify the combined invention of Wu, Frisken et al., and Kim by projecting 
a portion of each video object in a particular frame to intersect the projection of the 
video object in an adjacent frame to provide continuous silhouettes of the video object 
according to the time t coordinates as taught by Kumar to allow one in order to convert 
video frames into video mosaics, which is an efficient way of handling the large amount 
of data contained in the video stream. 

1 1 . Claim 4 is rejected under 35 U.S.C. 1 03(a) as being unpatentable over Wu (US 
6,529,202), Frisken et al. (US 6,603,484), Kim et al. (US 5,502,510) and Kumar et al. 
(PUB No.: US 2001/0038718) as applied to claim 3, further in view of Clatanoff et al. 
(US 5,592,231). 

Regarding claim 4, the combined invention of Wu, Frisken et al., Kim, and Kumar et al. 
teaches everything except for the following, which Clatanoff et al. teaches: 

• applying a spatial-domain 2D median filter to the frames to remove intensity 
singularities, without disturbing edge formation 

[Fig. 2, numeral 45; Col. 1, lines 65-67] 

Therefore, it would have been obvious to one of ordinary skill at the time the invention 
was made to modify the combined invention of Wu, Frisken et al., Kim, and Kumar et al. 
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by applying a spatial-domain 2D median filter to the frames as taught by Clatanoff et al. 
in order to remove impulse noise from the frames (as is well known in the art that the 
filter is well suited for). 

12. Claim 5 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wu (US 
6,529,202) and Frisken et al. (US 6,603,484) as applied to claims 1, 7, 12 further in view 
of Kim (US 5,978,031 ) and Gilhuijs et al. (US 6,1 12,1 12). 

Regarding claim 5, the combined invention of Wu and Frisken et al. teaches everything 
except the following, which Kim and Gilhuijs et al. teach: 

• partitioning the video into a plurality of identically sized volumes 
[Kim. Abstract: lines 1-4] 

• selecting the pixel at the center of each volume are the marker pixels 
[Gilhuijs etal. Col. 9, lines 23-29] 

Therefore, it would have been obvious to one of ordinary skill at the time the invention 
was made to modify the combined invention of Wu and Frisken et al. by also partitioning 
the video into a plurality of identically sized volumes (as taught by Kim) to facilitate 
block-based processing (e.g. each block can be a macroblock in MPEG encoding) and 
selecting the pixel at the center of each volume as the marker pixels (as taught by 
Gilhuis et al.) so as to have to fewest number of iterations to grow from a center pixel a 
region that is fully contained in the volume centered at this pixel. 
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13. Claims 8, 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over Wu 
(US 6,529,202) and Frisken et al. (US 6,603,484) as applied to claims 1, 7, 12 further in 
view of Rosenfeld et al. (Digital Picture Processing, 2""^ ed., Vol. 2, 1982, Sect. 10.4.2). 

Regarding claim 8, the combined invention of Wu and Frisken et al. teaches everything 
except the following, which Rosenfeld et al. teaches: 

• merging volumes less than minimum size with an adjacent volumes 

[Fig. 38 on P. 142; P. 142, lines 8-9. Note that a minimum size is implied by the 
statement "because C is small"] 

Therefore, it would have been obvious to one of ordinary skill at the time the invention 
was made to modify the combined invention of Wu and Frisken et al. by also merging 
volumes less than minimum size with an adjacent volumes as taught by Rosenfeld et al. 
in order to eliminate small "noise" volumes as Rosenfeld et al. points out in the last two 
lines of P. 141. 

Regarding claim 9, the combined invention of Wu, Frisken et al. and Rosenfeld et al. 
fails to expressly teach the following: 

• the minimum size is less than 0.001 of the volume representing the video 

However, at the time the invention was made, it would have been to a person of 
ordinary skill in the art to use a minimum size of less than 0.001 of the volume 
representing the video. Applicant has not disclosed that a minimum size of less than 
0.001 of the volume representing the video provides an advantage, is used for a 
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particular purpose, or solves a stated problem. One of ordinary skill in the art, 
furthermore, would have expected Applicant's invention to perform equally well with a 
minimum size of less than 0.002 of the volume representing the video because this will 
result in fewer volumes for subsequent processing and therefore can reduce processing 
time. 

Therefore, it would have been obvious to one of ordinary skill in this art to modify the 
combined invention of Wu, Frisken et al. and Rosenfeld et al. by using a minimum size 
of less than 0.001 of the volume representing the video to obtain the invention as 
specified in claim 9. 

14. Claim 10 is rejected under 35 U.S.C. 103(a) as being unpatentable over Wu (US 
6,529,202), Frisken et al. (US 6,603,484), and Rosenfeld et al, (Digital Picture 
Processing, 2""^ ed., Vol. 2, 1982, Sect. 10.4.2) as applied to claims 8, 9 above, further 
in view of McGee et al. (US 6.496,228) and Jewel (US 5,852,683). 

Regarding claim 10, the combined invention of Wu, Frisken et al. and Rosenfeld et al. 
fails to teach the following, which McGee et al. and Jewel teach: 

• sorting the volumes in an increasing order to size 
[McGee et al. Col. 13, lines 24-27] 

• processing the volumes in the increasing order 
[Jewel, Col. 6, lines 33-36] 
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In addition, the combined invention of Wu, Frisken et al. and Rosenfeld et al. further 
discloses: 

• the processing for each volume comprising: including each pixel of the volume in 
a closest volume until all volumes less than a minimum size are processed 
[Wu. Col. 4, lines 1-3, Note: In Wu's case the minimum size is inherently the 
size of the largest volume] 

Therefore, it would have been obvious to one of ordinary skill at the time the invention 
was made to modify the combined invention of Wu, Frisken et al. and Rosenfeld et al. 
by also sorting the volumes in an increasing order to size and processing the volumes in 
the increasing order as taught by McGee et al. and Jewel in order to easily identify 
smaller volumes to be merged into larger ones (as indicated in Rosenfeld et al., per 
analysis for claims 8 and 9) and ensure that the smallest ones get merged first. 

15. Claim 11 is rejected under 35 U.S.C. 103(a) as being unpatentable overWu (US 
6,529,202) and Frisken et al. (US 6,603,484) as applied to claims 1, 7, 12 further in view 
of Rudinetal. (US 6,452,637). 

Regarding claim 1 1, the combined invention of Wu and Frisken et al. teaches everything 
except for the following, which Rudin et al. teaches: 

• the descriptors include mutual descriptors of the volume and the adjacent volume 

[Col. 12, lines 40-49. Note that the merging scale is the mutual descriptor of 
volumes Rj and Rj] 
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Therefore, it would have been obvious to one of ordinary skill at the tinne the invention 
was made to modify the combined invention of Wu and Frisken et al. by also using a 
mutual descriptor as taught by Rudin et al. in order to have additional feature available 
for better merging decision. 

Allowable Subject Matter 

16. Claim 6 is objected to as being dependent upon a rejected base claim, but would 
be allowable if rewritten in independent form including all of the limitations of the base 
claim and any intervening claims. 

1 7. The following is a statement of reasons for the indication of allowable subject 
matter: 

Regarding claim 6, prior art of record fails to teach or suggest, alone or in combination a 
method for segmenting a video comprising, along with other limitations: 

• removing pixel in a predetermined neighborhood around the marker; and 
repeating the selecting and removing steps until no pixel remain. 

The closest art of Rosenberg (US 5.832,115) teaches 

• determining a gradient magnitude for each pixel in the video 
[Fig. 1, numeral 19; Fig. 2, numeral 19a; Col. 2, lines 44-47] 

• selecting the pixel with a minimum gradient magnitude as the marker pixel 
[Fig. 2, numeral 19b; Col. 2, lines 44-47] 
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In addition, Chang et al. (US 5,793,895) teaches an initial segmentation of an image 
into equal-sized macroblocks (i.e., a pre-determined neighborhood) [Col. 3, lines 39-41] 
and Pearson et al. (US 5,604,822) teaches starting segmentation from a seed point 
(e.g., a pixel with the smallest gradient) [Col. 12, lines 48-50]. However, they fail to 
teach segmentation by iteratively marking a pixel and removing a pre-determined 
neighborhood of that pixel. 



Contact Information 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Yubin Hung whose telephone number is (703) 305- 
1896. The examiner can normally be reached on 7:30 - 4:00. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Bhavesh Mehta can be reached on (703) 308-5246. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 



Yubin Hung 
Patent Examiner 
February 20, 2004 
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